APPARATUS AND METHOD FOR 
DETECTING ADAPTIVE MOTION DIRECTION 



BACKGROUND OF THE INVENTION 
5 Field of the Invention 

The present invention relates to an apparatus and 
method for detecting an adaptive motion direction of an image 
(object) . 

Description of the Related Art 
10 In a first prior art adaptive motion direction 

H detecting method, the following steps are carried out: 
g 1) first two-dimensional pixel data of an object 

ft! from a pickup camera for preset areas at time tl is fetched 

and stored in a first frame memory; 
O 15 2) second two-dimensional pixel data of the object 

^ from the pickup camera for the preset areas at time t2 (> tl) 
H is fetched and stored in a second frame memory; 
^ 3) differences between the first and second two- 

dimensional pixel data stored in the first and second frame 
jhf 20 memories are calculated; and 

4) an image of the object at time t3 (> t2) is . 
estimated to detect a motion direction of the contour of the 
object. 

The above-described first prior art adaptive motion 
25 direction detecting method is disclosed in JP-A-6-96210 and 
JP-8-44852. 

In the first prior art adaptive motion direction 
detecting method, however, since the image of the object has 
a bright contour and a dark contour which are generally 

30 different from each other, the calculated differences between 
the first and second two-dimensional pixel data depend upon 
the deviation between the bright contour and the dark contour, 
so that it is difficult to accurately determine the motion 
direction of the object. 

35 In the above-described first prior art adaptive 
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motion direction detecting method, in order to accurately 
determine the motion direction of an object independent of a 
bright contour and a dark contour thereof, two-dimensional 
pixel data in each of the present areas are normalized by their 
5 maximum lightness and their minimum lightness to form bright 
contour data and the dark contour data. Then, motion 
directions of the object are determined in accordance with the 
bright contour data and dark contour data, respectively. On 
the other hand, two-dimensional pixel data in each of the 
10 preset areas is caused to be binary data "1" when the lightness 
u.. thereof is larger than a definite lightness, and two- 
g dimensional pixel data in each of the preset areas is caused 
rj to be binary data "0" when the lightness thereof is not larger 

2 than the definite lightness. However, these techniques 
q 15 deteriorate the efficiency of motion detection. In addition, 
M the presence of the two frame memories does not save the 

L resource of memories. 

■i n 

N A second prior art adaptive motion direction 

m detecting method, spatial and time convoluting integrations 

O 20 are performed upon N-successive images for each local area at 
times t, t+At, t+2At, t+NAt, to obtain an optical flow, 
thus detecting a speed of an object (see: JP-A-7-192135) . 

In the above-described second prior art adaptive 
motion direction detecting method, however, since means for 
25 determining all gradients of the object are required, the 
resource thereof cannot be saved. That is, images 
statistically include horizontal displacement ; even in this 
case, spatial and tie convoluting interactions for all the 
directions have to be carried out, which decreases the 
30 efficiency of the resource thereof. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide 
an apparatus and method for detecting an adaptive motion 
35 direction capable of enhancing the resource efficiency and 
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saving the resource of memories. 

According to the present invention, in an adaptive 
motion direction detecting apparatus, an image input unit 
inputs two-dimensional pixel data of an object. A response 
5 output unit includes a plurality of response element arrays 
each having difference time phases, and each of the response 
element arrays includes a plurality of response elements. Each 
of the response elements generates a response output for one 
of a plurality of local areas partly superposed thereto. In 
10 this case, the two-dimensional pixel data is divided into the 
u local areas. A correlation function calculating unit 
D calculates spatial and time correlation functions between the 
S response outputs of the response elements. A response output 
Se selecting unit for selects response outputs of the response 
gl5 output unit for each of the local areas in accordance with the 
M spatial and time correlation functions. A motion direction 
J\ detecting unit includes a plural ity of detection elements each 
fjj corresponding to the response elements. Each of the detection 
J|: elements detects a motion direction of the object at one of 
Q20 the local areas in accordance with selected response outputs 
therefor. 

Also, in an adaptive motion direction detecting 
method, two-dimensional pixel data of an object is inputted. 
Then, a response output is generated for one of a plurality 

25 of local areas partly superposed thereon. In this case, the 
two-dimensional pixel data is divided into the local areas. 
Then, spatial and time correlation functions between the 
response outputs are calculated. Then, response outputs are 
selected for each of the local areas in accordance with the 

30 spatial and time correlation functions. Finally, a motion 
direction of the object at one of the local areas is detected 
in accordance with selected response outputs for the one of 
the local areas. 



35 



BRIEF DESCRIPTION OF THE DRAWINGS 
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The present invention will be more clearly 
understood from the description set forth below, as compared 
with the prior art, with reference to the accompanying 
drawings, wherein: 
5 Fig, 1 is a block circuit diagram illustrating an 

embodiment of the adaptive motion direction detecting 
apparatus according to the present invention; 

Fig. 2 is a diagram of pixel data inputted by the image 
input unit 1 of Fig, 1; 
10 Fig. 3 is a diagram of the response output unit of Fig. 

1; 

Fig. 4 is a diagram of the response selecting unit and 
the motion direction detecting unit of Fig. 1; 

Fig. 5A is a diagram illustrating an example of the output 
15 of one detection element of Fig. 4; 

Fig. 5B is a diagram of an arrow showing the gradient of 
the output of the detection element of Fig. 5A; 

Figs. 6 and 7 are diagrams of arrows showing examples of 
the gradients of the outputs of all the detection elements of 
20 Fig. 4; and 

Fig. 8 is a diagram of a memory into which the tables of 
Fig. 4 are combined. 



DESCRIPTION OF THE PREFERRED EMBODIMENT 
25 In Fig. 1, which illustrates an embodiment of the 

adaptive motion direction detecting apparatus according to 
the present invention, an image input unit 1, a response output 
unit 2, a correlation function calculating unit 3, a response 
selecting unit 4, a motion direction detecting unit 5, and an 
30 output unit 6 are provided. In this case, the response output 
unit 2, the correlation function calculating unit 3 and the 
response selecting unit 4 are constructed by a computer 
including a central processing unit (CPU), memories and the 
like. 

35 The image input unit 1 is constructed by a charge 
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coupled device (CCD) camera, for example, which inputs 
two-dimensional pixel data 101 such as 1000 x 1000 pixel data 
having coordinate k (k = 1 to 1000000) as shown in Fig, 2. In 
this case, pixel data of coordinate k at time t is represented 
5 by f k (t). 

The response output unit 2 is constructed by 
two-dimensional response element arrays 201, 202, 203 and 204 
each having 100 x 100 response elements as shown in Fig. 3. 
In this case, each of the response elements has coordinate m 
10 (m = 1 to 10000) . Also, each of the response element arrays 
u 201, 202, 203 and 204 has coordinate n (n = 1 to 4). 
O That is, the two-dimensional data area of the pixel 

y data is divided into local areas k iy A 2 , A m , A 10000 , which 
; y are partly superposed on each other. Then, each of the response 
gl5 elements m (m = 1, 2, — , 10000) receives pixel data f k (t) of 
Q the local area A B . Note that the local areas A ls A 2 , A B , 
— , A 10000 are realized by performing a Gaussian function upon 
fy the pixel data f k (t). 

[Jf The coordinate n corresponds to a response phase. 

p20 In this case, the difference in phase between the coordinate 
H n is 90° . 

A response function R Bt k> n (t) of the response 
element defined by coordinate m and coordinate n is 
represented by: 
25 R B , k> n (t) = S a . k • T n (t) (1) 

where S B> k (t) is a spatial response function; and 
T n (t) is a time response function. 
Also, the spatial response function S m> k (t) is 
represented by: 
30 S a> k 

= 1/(2tt A ex 2 ) • exp(-d n , k 2 /(2A ex 2 )) 
-k/(2tc A inh 2 ) • exp(-d m> k 7(2A inh 2 )) (2) 
where A 6X andA inh are coefficients for showing 
spatial spread of the pixel data for one response element 
35 (A ex < A inh , for example, A ex : A inh = 0.25 : 1); and 
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d m> k is a distance between the coordinate m and the 
coordinate k. 

Note that the coordinate m is defined by a center 
of the local area A„of the coordinate k (see: dots of the pixel 
data 101 of Fig. 3). Therefore, the distance d n> k is 
represented by: 

d B> k 2 = (x B - x k ) 2 + (y B - y k ) 2 (3) 

where x„ and y„ are an X-direction value and a Y- 
direction value, respectively, of the coordinate m; and 

x k and y k are an X-direction value and a Y-direction 
value, respectively, of the coordinate k. 

The time response function T„ (t) is represented 

by : 

T, (t) = exp(-t/A t ) • sin(cot) 

• © Ct(T 0 -t) ] (4) 

T 2 (t) = -exp(-(T 0 + r 0 - t)/A t ) • sin [w(t-r 0 )3 

• 0 [(t-r 0 ) (T 0 +r,-t)3 ' (5) 
T 3 (t) = -T, (t) (6) 
T 4 (t) = -T, (t) (7) 
whereA t is a coefficient for showing a spread of 

time response; 

T 0 is a period of the time response; 
r„ is a delay time; 

to is an angular frequency (=27r/T 0 ); and 
0 is a step function. 

The response element at coordinate m and coordinate 
n generates the following response output Vn.nit): 

*7„,„(t) = fdf ZE a , k . a (f)- f k (t - V) (8) 
The correlation function calculating unit 3 
calculates a correlation function r,, n . „. between the 
response output v «, n (t) of a response element having 
coordinate m and coordinate n and the response output tj B . in .(t) 
of a response element having coordinate m' and coordinate n' 
by: 

r„, n; ... „-=<77 m> „(t) • v ... ».(t)> t 
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- fdt?7...<t) ■ 77... B -(t) (9) 
where < > t means an average value with respect 

to time. 

The response selecting unit 4 selects response 
5 outputs having high spatial and time correlations from the 
response outputs 7? m , n (t) of the response output unit 2 in 
accordance with the correlation functions F Bi n . B . j n > 
calculated by the correlation function calculating unit 3. 
Also, the motion direction detecting unit 5 detects a motion 
10 direction of an image in accordance with the output of the 
response selecting unit 4. 
M As illustrated in Fig. 4, the motion direction 

^ detecting unit 5 is constructed by a two-dimensional detection 
fjj element array having 100 x 100 detection elements having 

coordinate i. Also, the response selecting unit 4 is 
5 constructed by tables 401, 40i' , — for storing o\. < m n 
:% * corresponding to the detection elements of coordinate i (i = 
^ 1 to 10000). 

J The response selecting unit 4 is operated 

so that the following cost function E is minimum: 

D E = JEVj.pE,. (10) 

E i' = "i,?.^'^ 1 ' j> B - n ' ° V ••>"' »'• n ' r ra > n 5 ■'■ »' 
M.i' 

= l/(2;r A, 2 ) • exp(-d i(i . 2 /(2A, 2 )) 
25 -1/(2ttA 2 2 ) • exp(-d iii ,V(2A 2 2 )) (12) 

where A, and A 2 are coefficients (A ex < A inh , for 
example, A ex : A inh = 0.25 : 1); and • 

d i( }. is a distance between the coordinate i and the 
coordinate i'. 

30 Note that the coordinates i and i' are defined by 

a center of the local area of the coordinates i and i', 
respectively. Therefore, the distance d it ,. is represented by: 
di, = (x, - x r ) 2 + (y, - yi .) 2 (13) 
where Xj and y t are an X-direction value and a 
35 Y-direction value, respectively, of the coordinate i. 
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x r and y v are an X-direction value and a Y- 
direction value, respectively, of the coordinate i'; 

Also, the output r? i ( t ) of the detection element of 
coordinate i is represented by: 

7?i(t) = 2R i(k (t) • f k (t) < (14) 

R i( k (t) = ga lt j. „, n • R„, k> n (t) (15) 
Note that, only when the response output v B> „(t) 
from the response output unit 2 is received by the response 
selecting unit 4, is cr it it B> n "1". Otherwise, a u St B> „ is "0". 

Here, consider that f k (t) is a grating pattern or 
a sinusoidal wave-shaped gray pattern given by: 
f k (t | 0, f s , f t ) 

= cos [27Tf s (x k cos0+ y k sin0 )-27tf t • t] (16) 
where 6 is a gradient of the grating pattern 

corresponding to an angle of a motion direction of an object; 

f s is a space frequency of the grating pattern; and 
f t is a time frequency of the grating pattern. In 

this case, the formula (14) is replaced by: 
VM \ 0, f s , f t ) 

= £dt'2R i(k (t') • f k (t' - t | 6, f„ f t ) (17) 

In the formula (17), the value of 6 gives a 
direction selectivity of the detection element of coordinate 
i when7?i(t | 6 , f s , f t ) is maximum. In other words, if Q does 
not show the direction selectivity of the detection element 
of coordinate i , t? £ ( t | 6, f s , f t ) is very small and can be 
negligible. Thus, if pixel data f k (t) is given, the motion 
direction of an image included in the pixel data f k (t) can be 
detected by the output t? t ( t | 9 , f 8 , f t ) of the detection 
elements having coordinate i. 

The operation of the adaptive motion direction 
detecting apparatus of Fig. 1 is explained simply below. 

First, the image input unit 1 receives pixel data 
101 as shown in Fig. 2. In this case, each of pixel data 101 
is a gray signal. 

The response output unit 2 divides two-dimensional 
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data area of the pixel data into a plurality of local areas 
An A 2 , A k , — as shown in Fig. 3. Then, the response output 
unit 2 generates the response output 7? ffl>n (t) for each of the 
local areas A t , A 2 , — , A k , •••• 
5 The correlation function calculating unit 3 

calculates spatial and time correlation functions r a> n . B . f n . 
between the response outputs of the response output unit 2. 

The response selecting unit 4 selects response 
outputs 77 m)n (t) of the response outputs unit 2 in accordance 
10 with the spatial and time correlation function T m> n; m > 9 n .. 

The motion direction detecting unit 5 detects a 
£!; motion direction in accordance with the selection result, 
O cr iijf fflj n of the response selecting unit 4 and the response 

J output 77 B>n (t) of the response output unit 2. 

0 15 In Fig. 5A, which shows an example of the output 
Jj*5 7?i(t) of one detection element of coordinate i of Fig. 4, 

depicted on the output unit 6, a solid line shows a bright 
ji: contour of an object, while a dotted line shows a dark contour 
ry of the object. In this case, the bright contour and the dark 

1 20 contour are both moving along the same direction of a gradient 
§1 0 as indicated by Fig. 5B. 

Figs. 6 and 7 are diagrams showing examples of 
arrows of the gradients of the outputs of all the detection 
elements of Fig. 4 where the number of detection elements is 

25 50 x 50. Particularly, Fig. 7 shows a case where an object 
is moving around 0° and 180° . 

In the above-described embodiments, when an object 
(image) is moving, the number of a itL ffli n (="1") selected by 
the response selecting unit 4 is increased. On the other hand, 

30 when an object (image) is not moving, the number of cr i>j( m> 
n (="1") selected by the response selecting unit 4 is decreased. 
Note that the tables 401, — , 40i' , — of Fig. 4 are used for 
storing cr^j^ m> n regardless of its value "1" or "0"; however, 
the tables 401, 40T , ••• can be combined into a memory as 

35 shown in Fig. 8 which stores only coordinates i, j, m and n 
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of o' itj> Bi n whose value is "1". The memory of Fig. 8 can be 
reduced in size as compared with all of the tables 401, 
40 i , — of Fig. 4, which is helpful in saving the resource of 
memories. 

In the above-described embodiment, when the 
response element arrays 201, 202, 203 and 204 are constructed 
so as to have a response time phase of 90° , the detection 
elements of the motion direction detecting unit 5 are operated 
in accordance with the response outputs of the response 
element arrays 201, 202, 203 and 204 to obtain a motion 
direction angle from 0° to 360° as shown in Fig. 5A. This has 
been mathematically proved by the formulae (1) to (7). 

Also, in the above-described embodiment, since one 
two-dimensional pixel data area is divided into partly- 
superposed local areas A 1? A 2 , — , A m , — as shown in Fig. 3, 
the response output of the response elements, of one local area 
includes a part of the response output of the response element 
of its adjacent local areas, which is helpful in enhancing the 
detection accuracy of motion directions. 

In the prior art adaptive motion direction 
apparatus where two frame memories are used for storing image 
data at different timings, the frame memories require a 
capacity of 1000 x 1000 x 2 bits (=2,000,000 bits). On the 
other hand, in the above-described embodiment, 

1000 x 1000 x 1 bits (image input unit), 
100 x 100 x 4 bits (response elements), 
100 x 100 x 1 bits (detection elements) and, 
100 x 100 x 20 bits (memory of Fig. 8) are required, 
i.e., only 1,250,000 bits are required. Thus, a remarkable 
reduction of memories can be expected. 

Also, since it is unnecessary to compare two kinds 
of image data to detect a motion direction of an object (image), 
the efficiency of detection of a motion direction can be 
enhanced. 

As explained hereinabove, according to the present 
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invention, a motion direction of an object (image) can be 
detected independent of the object whose contour being bright 
or dark. Also, the detection resource efficiency of an 
accurate motion direction can be enhanced. Further, the 
detected motion direction of an object is statistically 
reflected by the motion of an input image of the object. 
Additionally, the resource of memories can be saved. 



